{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SRL.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzRpKWxSgZmb"
      },
      "source": [
        "#Semantic Role Labeling(SRL)\n",
        "\n",
        "The notebook is an implementation of the Allen Institute for AI BERT-based model. [Reference usage of the Notebook](https://demo.allennlp.org/semantic-role-labeling/MjE4NjI1Ng==)\n",
        "\n",
        "The BERT-based model is an implementation of [Peng Shi and Jimmy Lin, (2019), ‘Simple BERT Models for Relation Extraction and Semantic Role Labeling’]( https://arxiv.org/abs/1904.05255)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9aeqrxgQhKmE"
      },
      "source": [
        "Intalling Allen NLP"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XAIkwYFaeBBD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d75870e6-02e6-4d6f-f165-c89bf32e17bb"
      },
      "source": [
        "!pip install allennlp==1.0.0 allennlp-models==1.0.0"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting allennlp==1.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/2c/49/bf0ec241496a82c9dd2f0b6ff6f8156b6b2b72b849df8c00a4f2bcf61485/allennlp-1.0.0-py3-none-any.whl (473kB)\n",
            "\u001b[K     |████████████████████████████████| 481kB 9.4MB/s \n",
            "\u001b[?25hCollecting allennlp-models==1.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d5/9ee1d0b8c217b6978e42e54fbab8bafe9e792f0f8262f381dde44cee44ae/allennlp_models-1.0.0-py3-none-any.whl (282kB)\n",
            "\u001b[K     |████████████████████████████████| 286kB 28.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.19 in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (4.41.1)\n",
            "Requirement already satisfied: spacy<2.3,>=2.1.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (2.2.4)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (0.22.2.post1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (1.4.1)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (3.2.5)\n",
            "Collecting torch<1.6.0,>=1.5.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/01/457b49d790b6c4b9720e6f9dbbb617692f6ce8afdaadf425c055c41a7416/torch-1.5.1-cp36-cp36m-manylinux1_x86_64.whl (753.2MB)\n",
            "\u001b[K     |████████████████████████████████| 753.2MB 23kB/s \n",
            "\u001b[?25hRequirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (3.6.4)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (0.8)\n",
            "Collecting overrides==3.0.0\n",
            "  Downloading https://files.pythonhosted.org/packages/42/8d/caa729f809ecdf8e76fac3c1ff7d3f0b72c398c9dd8a6919927a30a873b3/overrides-3.0.0.tar.gz\n",
            "Requirement already satisfied: requests>=2.18 in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (2.23.0)\n",
            "Collecting jsonpickle\n",
            "  Downloading https://files.pythonhosted.org/packages/ee/d5/1cc282dc23346a43aab461bf2e8c36593aacd34242bee1a13fa750db0cfe/jsonpickle-1.4.2-py2.py3-none-any.whl\n",
            "Collecting transformers<2.12,>=2.9\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/48/35/ad2c5b1b8f99feaaf9d7cdadaeef261f098c6e1a6a2935d4d07662a6b780/transformers-2.11.0-py3-none-any.whl (674kB)\n",
            "\u001b[K     |████████████████████████████████| 675kB 42.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock<3.1,>=3.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (3.0.12)\n",
            "Collecting tensorboardX>=1.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/af/0c/4f41bcd45db376e6fe5c619c01100e9b7531c55791b7244815bac6eac32c/tensorboardX-2.1-py2.py3-none-any.whl (308kB)\n",
            "\u001b[K     |████████████████████████████████| 317kB 30.5MB/s \n",
            "\u001b[?25hCollecting boto3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f6/bf/6fd00f2e8b2f9e8688b10b616c66be985a0053729cb1e92ac2f6e8ec1cd6/boto3-1.16.40-py2.py3-none-any.whl (130kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 47.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (1.19.4)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from allennlp==1.0.0) (2.10.0)\n",
            "Collecting jsonnet>=0.10.0; sys_platform != \"win32\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/40/6f16e5ac994b16fa71c24310f97174ce07d3a97b433275589265c6b94d2b/jsonnet-0.17.0.tar.gz (259kB)\n",
            "\u001b[K     |████████████████████████████████| 266kB 59.0MB/s \n",
            "\u001b[?25hCollecting word2number>=1.1\n",
            "  Downloading https://files.pythonhosted.org/packages/4a/29/a31940c848521f0725f0df6b25dca8917f13a2025b0e8fcbe5d0457e45e6/word2number-1.1.zip\n",
            "Collecting conllu==3.0\n",
            "  Downloading https://files.pythonhosted.org/packages/66/0b/a8863b5c14aee200a13a0f8c28550fd0132e947ae88441c9f517eb84613b/conllu-3.0-py2.py3-none-any.whl\n",
            "Collecting py-rouge==1.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9c/1d/0bdbaf559fb7afe32308ebc84a2028600988212d7eb7fb9f69c4e829e4a0/py_rouge-1.1-py3-none-any.whl (56kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 5.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (7.4.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (3.0.5)\n",
            "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (1.0.5)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (2.0.5)\n",
            "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (1.1.3)\n",
            "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (0.4.1)\n",
            "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (1.0.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (50.3.2)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (0.8.0)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.3,>=2.1.0->allennlp==1.0.0) (1.0.5)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->allennlp==1.0.0) (1.0.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk->allennlp==1.0.0) (1.15.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch<1.6.0,>=1.5.0->allennlp==1.0.0) (0.16.0)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==1.0.0) (20.3.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==1.0.0) (8.6.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==1.0.0) (0.7.1)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==1.0.0) (1.10.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==1.0.0) (1.4.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==1.0.0) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==1.0.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==1.0.0) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==1.0.0) (1.24.3)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from jsonpickle->allennlp==1.0.0) (3.3.0)\n",
            "Collecting tokenizers==0.7.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/e5/a26eb4716523808bb0a799fcfdceb6ebf77a18169d9591b2f46a9adb87d9/tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8MB 48.9MB/s \n",
            "\u001b[?25hCollecting sentencepiece\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/2d/6d4ca4bef9a67070fa1cac508606328329152b1df10bdf31fb6e4e727894/sentencepiece-0.1.94-cp36-cp36m-manylinux2014_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 47.5MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 50.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers<2.12,>=2.9->allennlp==1.0.0) (2019.12.20)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers<2.12,>=2.9->allennlp==1.0.0) (20.8)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorboardX>=1.2->allennlp==1.0.0) (3.12.4)\n",
            "Collecting jmespath<1.0.0,>=0.7.1\n",
            "  Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n",
            "Collecting botocore<1.20.0,>=1.19.40\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ea/77/13fc099a10c22d08d766e244412c114694e21982c04cafc1ade33d8a430c/botocore-1.19.40-py2.py3-none-any.whl (7.1MB)\n",
            "\u001b[K     |████████████████████████████████| 7.1MB 34.3MB/s \n",
            "\u001b[?25hCollecting s3transfer<0.4.0,>=0.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)\n",
            "\u001b[K     |████████████████████████████████| 71kB 7.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->jsonpickle->allennlp==1.0.0) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->jsonpickle->allennlp==1.0.0) (3.4.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers<2.12,>=2.9->allennlp==1.0.0) (7.1.2)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers<2.12,>=2.9->allennlp==1.0.0) (2.4.7)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.20.0,>=1.19.40->boto3->allennlp==1.0.0) (2.8.1)\n",
            "Building wheels for collected packages: overrides, jsonnet, word2number, sacremoses\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for overrides: filename=overrides-3.0.0-cp36-none-any.whl size=5669 sha256=6f80143088a78455dd287d78fdced3dd986f9bb4a23edb94ec3376db1b81df6f\n",
            "  Stored in directory: /root/.cache/pip/wheels/6f/1b/ec/6c71a1eb823df7f850d956b2d8c50a6d49c191e1063d73b9be\n",
            "  Building wheel for jsonnet (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for jsonnet: filename=jsonnet-0.17.0-cp36-cp36m-linux_x86_64.whl size=3387942 sha256=1418eb22c110c8535ec13a7e223704f7d00de2f9d27b395203bcc0a6044e03aa\n",
            "  Stored in directory: /root/.cache/pip/wheels/26/7a/37/7dbcc30a6b4efd17b91ad1f0128b7bbf84813bd4e1cfb8c1e3\n",
            "  Building wheel for word2number (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for word2number: filename=word2number-1.1-cp36-none-any.whl size=5588 sha256=6c711b492080c629e45bd6f594bbd082f84b77fb2859e691b07f8cc43e891868\n",
            "  Stored in directory: /root/.cache/pip/wheels/46/2f/53/5f5c1d275492f2fce1cdab9a9bb12d49286dead829a4078e0e\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893261 sha256=ca22850c9a27802373c980ccdce43020db3a1c30576474ebc27c849dd7a8374e\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built overrides jsonnet word2number sacremoses\n",
            "\u001b[31mERROR: torchvision 0.8.1+cu101 has requirement torch==1.7.0, but you'll have torch 1.5.1 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: botocore 1.19.40 has requirement urllib3<1.27,>=1.25.4; python_version != \"3.4\", but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n",
            "Installing collected packages: torch, overrides, jsonpickle, tokenizers, sentencepiece, sacremoses, transformers, tensorboardX, jmespath, botocore, s3transfer, boto3, jsonnet, allennlp, word2number, conllu, py-rouge, allennlp-models\n",
            "  Found existing installation: torch 1.7.0+cu101\n",
            "    Uninstalling torch-1.7.0+cu101:\n",
            "      Successfully uninstalled torch-1.7.0+cu101\n",
            "Successfully installed allennlp-1.0.0 allennlp-models-1.0.0 boto3-1.16.40 botocore-1.19.40 conllu-3.0 jmespath-0.10.0 jsonnet-0.17.0 jsonpickle-1.4.2 overrides-3.0.0 py-rouge-1.1 s3transfer-0.3.3 sacremoses-0.0.43 sentencepiece-0.1.94 tensorboardX-2.1 tokenizers-0.7.0 torch-1.5.1 transformers-2.11.0 word2number-1.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcSZJu8ohUv5"
      },
      "source": [
        "Sample 1: Did Bob really think he could prepare a meal for 50 people in only a few hours?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1pziWuZKeMti",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cef0cbfb-9256-45d6-9419-3cba6bd616c3"
      },
      "source": [
        "!echo '{\"sentence\": \"Did Bob really think he could prepare a meal for 50 people in only a few hours?\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:07:37,371 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:07:37.529750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:07:39,325 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
            "2020-12-20 09:07:43,159 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:07:43,159 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:07:43,160 - INFO - filelock - Lock 140370276310488 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:07:43,161 - INFO - allennlp.common.file_utils - https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz not found in cache, downloading to /root/.allennlp/cache/tmpqkld8r7a.tmp\n",
            "100% 406056588/406056588 [00:06<00:00, 64834243.25B/s]\n",
            "2020-12-20 09:07:49,714 - INFO - allennlp.common.file_utils - Renaming temp file /root/.allennlp/cache/tmpqkld8r7a.tmp to cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:07:49,714 - INFO - allennlp.common.file_utils - creating metadata file for /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:07:49,715 - INFO - filelock - Lock 140370276310488 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:07:49,715 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:07:49,715 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpaqcbgixa\n",
            "2020-12-20 09:07:54,008 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:07:54,009 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpaqcbgixa/vocabulary.\n",
            "2020-12-20 09:07:54,009 - INFO - filelock - Lock 140370276158376 acquired on /tmp/tmpaqcbgixa/vocabulary/.lock\n",
            "2020-12-20 09:07:54,036 - INFO - filelock - Lock 140370276158376 released on /tmp/tmpaqcbgixa/vocabulary/.lock\n",
            "2020-12-20 09:07:54,037 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:07:54,037 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:07:54,037 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:07:54,037 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:07:54,038 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7faa80ddc7b8>\n",
            "2020-12-20 09:07:54,038 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:07:54,038 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:07:54,038 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:07:54,338 - INFO - filelock - Lock 140370267846920 acquired on /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517.lock\n",
            "2020-12-20 09:07:54,339 - INFO - transformers.file_utils - https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmp9z3yyeli\n",
            "Downloading: 100% 433/433 [00:00<00:00, 297kB/s]\n",
            "2020-12-20 09:07:54,651 - INFO - transformers.file_utils - storing https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json in cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:07:54,651 - INFO - transformers.file_utils - creating metadata file for /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:07:54,651 - INFO - filelock - Lock 140370267846920 released on /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517.lock\n",
            "2020-12-20 09:07:54,652 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:07:54,652 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:07:54,878 - INFO - filelock - Lock 140370267847312 acquired on /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157.lock\n",
            "2020-12-20 09:07:54,879 - INFO - transformers.file_utils - https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmp2oj6r8r1\n",
            "Downloading: 100% 440M/440M [00:10<00:00, 43.8MB/s]\n",
            "2020-12-20 09:08:04,981 - INFO - transformers.file_utils - storing https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin in cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:04,981 - INFO - transformers.file_utils - creating metadata file for /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:04,982 - INFO - filelock - Lock 140370267847312 released on /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157.lock\n",
            "2020-12-20 09:08:04,982 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:07,795 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:08:07,796 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,797 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,798 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:08:07,799 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:08:07,800 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,801 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:08:07,802 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:08:07,803 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:08:07,804 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:08:07,805 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:08:07,806 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:08:07,807 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:08:08,259 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:08:08,260 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:08:08,568 - INFO - filelock - Lock 140370276307520 acquired on /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock\n",
            "2020-12-20 09:08:08,568 - INFO - transformers.file_utils - https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmp733mj1ki\n",
            "Downloading: 100% 232k/232k [00:00<00:00, 880kB/s]\n",
            "2020-12-20 09:08:09,117 - INFO - transformers.file_utils - storing https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt in cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "2020-12-20 09:08:09,117 - INFO - transformers.file_utils - creating metadata file for /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "2020-12-20 09:08:09,117 - INFO - filelock - Lock 140370276307520 released on /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock\n",
            "2020-12-20 09:08:09,118 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"Did Bob really think he could prepare a meal for 50 people in only a few hours?\"}\n",
            "prediction:  {\"verbs\": [{\"verb\": \"think\", \"description\": \"Did [ARG0: Bob] [ARGM-ADV: really] [V: think] [ARG1: he could prepare a meal for 50 people in only a few hours] ?\", \"tags\": [\"O\", \"B-ARG0\", \"B-ARGM-ADV\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\"]}, {\"verb\": \"could\", \"description\": \"Did Bob really think he [V: could] prepare a meal for 50 people in only a few hours ?\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"B-V\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"prepare\", \"description\": \"Did Bob really think [ARG0: he] [ARGM-MOD: could] [V: prepare] [ARG1: a meal for 50 people] [ARGM-TMP: in only a few hours] ?\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"B-ARGM-MOD\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"B-ARGM-TMP\", \"I-ARGM-TMP\", \"I-ARGM-TMP\", \"I-ARGM-TMP\", \"I-ARGM-TMP\", \"O\"]}], \"words\": [\"Did\", \"Bob\", \"really\", \"think\", \"he\", \"could\", \"prepare\", \"a\", \"meal\", \"for\", \"50\", \"people\", \"in\", \"only\", \"a\", \"few\", \"hours\", \"?\"]}\n",
            "\n",
            "2020-12-20 09:08:10,277 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpaqcbgixa\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hWHpOrNvZQ3m"
      },
      "source": [
        "Sample 2: Mrs. and Mr. Tomaso went to Europe for vacation and visited Paris and first went to visit the Eiffel Tower."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFKPLyqihrB_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0f52798-903a-4d04-f388-d26a700b8cd7"
      },
      "source": [
        "!echo '{\"sentence\": \"Mrs. and Mr. Tomaso went to Europe for vacation and visited Paris and first went to visit the Eiffel Tower.\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:08:12,622 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:08:12.774532: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:08:14,547 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:08:15,761 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:15,761 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:15,762 - INFO - filelock - Lock 139888470380440 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:15,763 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:08:15,763 - INFO - filelock - Lock 139888470380440 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:15,763 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:15,763 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmp7jeuj77a\n",
            "2020-12-20 09:08:19,975 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:08:19,976 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmp7jeuj77a/vocabulary.\n",
            "2020-12-20 09:08:19,976 - INFO - filelock - Lock 139888468747992 acquired on /tmp/tmp7jeuj77a/vocabulary/.lock\n",
            "2020-12-20 09:08:20,002 - INFO - filelock - Lock 139888468747992 released on /tmp/tmp7jeuj77a/vocabulary/.lock\n",
            "2020-12-20 09:08:20,003 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7f3a52797748>\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:08:20,004 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:08:20,298 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:08:20,298 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:08:20,492 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:23,170 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:08:23,171 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:08:23,171 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:08:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:08:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:08:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:08:23,179 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:08:23,180 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:08:23,208 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,209 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,210 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:08:23,211 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:08:23,212 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:08:23,664 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:08:23,664 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:08:23,665 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:08:23,987 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"Mrs. and Mr. Tomaso went to Europe for vacation and visited Paris and first went to visit the Eiffel Tower.\"}\n",
            "prediction:  {\"verbs\": [{\"verb\": \"went\", \"description\": \"[ARG0: Mrs. and Mr. Tomaso] [V: went] [ARG4: to Europe] [ARGM-PRP: for vacation] and visited Paris and first went to visit the Eiffel Tower .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"B-V\", \"B-ARG4\", \"I-ARG4\", \"B-ARGM-PRP\", \"I-ARGM-PRP\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"visited\", \"description\": \"[ARG0: Mrs. and Mr. Tomaso] went to Europe for vacation and [V: visited] [ARG1: Paris] and first went to visit the Eiffel Tower .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"went\", \"description\": \"[ARG0: Mrs. and Mr. Tomaso] went to Europe for vacation and visited Paris and [ARGM-TMP: first] [V: went] [ARGM-PRP: to visit the Eiffel Tower] .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARGM-TMP\", \"B-V\", \"B-ARGM-PRP\", \"I-ARGM-PRP\", \"I-ARGM-PRP\", \"I-ARGM-PRP\", \"I-ARGM-PRP\", \"O\"]}, {\"verb\": \"visit\", \"description\": \"[ARG0: Mrs. and Mr. Tomaso] went to Europe for vacation and visited Paris and first went to [V: visit] [ARG1: the Eiffel Tower] .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\"]}], \"words\": [\"Mrs.\", \"and\", \"Mr.\", \"Tomaso\", \"went\", \"to\", \"Europe\", \"for\", \"vacation\", \"and\", \"visited\", \"Paris\", \"and\", \"first\", \"went\", \"to\", \"visit\", \"the\", \"Eiffel\", \"Tower\", \".\"]}\n",
            "\n",
            "2020-12-20 09:08:25,342 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmp7jeuj77a\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v45ooI5ReoXk"
      },
      "source": [
        "Sample 3:John wanted to drink tea, Mary likes to drink coffee but Karim drank some cool water and Faiza would like to drink tomato juice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pz-jLVeAersa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "13b50208-4caf-476d-abf8-627b21c65863"
      },
      "source": [
        "!echo '{\"sentence\": \"John wanted to drink tea, Mary likes to drink coffee but Karim drank some cool water and Faiza would like to drink tomato juice.\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:08:27,582 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:08:27.767124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:08:29,592 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:08:30,797 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:30,797 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:30,799 - INFO - filelock - Lock 140584440236464 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:30,799 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:08:30,799 - INFO - filelock - Lock 140584440236464 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:30,799 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:30,799 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpse7z902p\n",
            "2020-12-20 09:08:35,061 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:08:35,061 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpse7z902p/vocabulary.\n",
            "2020-12-20 09:08:35,062 - INFO - filelock - Lock 140584442130328 acquired on /tmp/tmpse7z902p/vocabulary/.lock\n",
            "2020-12-20 09:08:35,089 - INFO - filelock - Lock 140584442130328 released on /tmp/tmpse7z902p/vocabulary/.lock\n",
            "2020-12-20 09:08:35,089 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7fdc5d9b87b8>\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:08:35,090 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:08:35,400 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:08:35,400 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:08:35,598 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:38,288 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:08:38,289 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:08:38,289 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:08:38,289 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:08:38,289 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,290 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:08:38,291 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:08:38,292 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:08:38,293 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,294 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,295 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:08:38,296 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,374 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,375 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:08:38,376 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:08:38,830 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:08:38,831 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:08:39,125 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"John wanted to drink tea, Mary likes to drink coffee but Karim drank some cool water and Faiza would like to drink tomato juice.\"}\n",
            "prediction:  {\"verbs\": [{\"verb\": \"wanted\", \"description\": \"[ARG0: John] [V: wanted] [ARG1: to drink tea] , Mary likes to drink coffee but Karim drank some cool water and Faiza would like to drink tomato juice .\", \"tags\": [\"B-ARG0\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"drink\", \"description\": \"[ARG0: John] wanted to [V: drink] [ARG1: tea] , Mary likes to drink coffee but Karim drank some cool water and Faiza would like to drink tomato juice .\", \"tags\": [\"B-ARG0\", \"O\", \"O\", \"B-V\", \"B-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"likes\", \"description\": \"John wanted to drink tea , [ARG0: Mary] [V: likes] [ARG1: to drink coffee] but Karim drank some cool water and Faiza would like to drink tomato juice .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"drink\", \"description\": \"John wanted to drink tea , [ARG0: Mary] likes to [V: drink] [ARG1: coffee] but Karim drank some cool water and Faiza would like to drink tomato juice .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"O\", \"O\", \"B-V\", \"B-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"drank\", \"description\": \"John wanted to drink tea , Mary likes to drink coffee but [ARG0: Karim] [V: drank] [ARG1: some cool water and Faiza] would like to drink tomato juice .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"would\", \"description\": \"John wanted to drink tea , Mary likes to drink coffee but Karim drank some cool water and Faiza [V: would] [ARGM-DIS: like] to drink tomato juice .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARGM-DIS\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"like\", \"description\": \"John wanted to drink tea , Mary likes to drink coffee but Karim drank [ARG0: some cool water and Faiza] [ARGM-MOD: would] [V: like] [ARG1: to drink tomato juice] .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"B-ARGM-MOD\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\"]}, {\"verb\": \"drink\", \"description\": \"John wanted to drink tea , Mary likes to drink coffee but Karim drank [ARG0: some cool water and Faiza] would like to [V: drink] [ARG1: tomato juice] .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"O\"]}], \"words\": [\"John\", \"wanted\", \"to\", \"drink\", \"tea\", \",\", \"Mary\", \"likes\", \"to\", \"drink\", \"coffee\", \"but\", \"Karim\", \"drank\", \"some\", \"cool\", \"water\", \"and\", \"Faiza\", \"would\", \"like\", \"to\", \"drink\", \"tomato\", \"juice\", \".\"]}\n",
            "\n",
            "2020-12-20 09:08:40,852 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpse7z902p\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7QVm45YmxTt"
      },
      "source": [
        "Sample 4: Alice, whose husband went jogging every Sunday, liked to go to a dancing class in the meantime."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mvm6zN7_m0GI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "736f6e5d-0f14-4692-962d-696f3e884c5e"
      },
      "source": [
        "!echo '{\"sentence\": \"Alice, whose husband went jogging every Sunday, liked to go to a dancing class in the meantime.\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:08:43,153 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:08:43.324358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:08:45,080 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:08:46,294 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:46,294 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:46,295 - INFO - filelock - Lock 139693451663232 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:46,295 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:08:46,295 - INFO - filelock - Lock 139693451663232 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:08:46,295 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:08:46,295 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpmnefy_l0\n",
            "2020-12-20 09:08:50,552 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:08:50,552 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpmnefy_l0/vocabulary.\n",
            "2020-12-20 09:08:50,552 - INFO - filelock - Lock 139692801159064 acquired on /tmp/tmpmnefy_l0/vocabulary/.lock\n",
            "2020-12-20 09:08:50,580 - INFO - filelock - Lock 139692801159064 released on /tmp/tmpmnefy_l0/vocabulary/.lock\n",
            "2020-12-20 09:08:50,580 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7f0cc3c6a7b8>\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:08:50,581 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:08:50,888 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:08:50,889 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:08:50,928 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:08:53,601 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:08:53,602 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,603 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,604 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:08:53,605 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:08:53,606 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:08:53,607 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,608 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:08:53,609 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:08:53,683 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:08:53,684 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:08:53,685 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:08:53,686 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:08:53,686 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:08:54,186 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:08:54,187 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:08:54,497 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"Alice, whose husband went jogging every Sunday, liked to go to a dancing class in the meantime.\"}\n",
            "prediction:  {\"verbs\": [{\"verb\": \"went\", \"description\": \"Alice , [ARG1: whose husband] [V: went] [ARG2: jogging] [ARGM-TMP: every Sunday] , liked to go to a dancing class in the meantime .\", \"tags\": [\"O\", \"O\", \"B-ARG1\", \"I-ARG1\", \"B-V\", \"B-ARG2\", \"B-ARGM-TMP\", \"I-ARGM-TMP\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"jogging\", \"description\": \"Alice , [ARG0: whose husband] went [V: jogging] [ARGM-TMP: every Sunday] , liked to go to a dancing class in the meantime .\", \"tags\": [\"O\", \"O\", \"B-ARG0\", \"I-ARG0\", \"O\", \"B-V\", \"B-ARGM-TMP\", \"I-ARGM-TMP\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\"]}, {\"verb\": \"liked\", \"description\": \"[ARG0: Alice , whose husband went jogging every Sunday] , [V: liked] [ARG1: to go to a dancing class in the meantime] .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"B-V\", \"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\"]}, {\"verb\": \"go\", \"description\": \"[ARG0: Alice , whose husband went jogging every Sunday] , liked to [V: go] [ARG4: to a dancing class] [ARGM-TMP: in the meantime] .\", \"tags\": [\"B-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"I-ARG0\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARG4\", \"I-ARG4\", \"I-ARG4\", \"I-ARG4\", \"B-ARGM-TMP\", \"I-ARGM-TMP\", \"I-ARGM-TMP\", \"O\"]}, {\"verb\": \"dancing\", \"description\": \"Alice , whose husband went jogging every Sunday , liked to go to a [V: dancing] [ARG0: class] in the meantime .\", \"tags\": [\"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"O\", \"B-V\", \"B-ARG0\", \"O\", \"O\", \"O\", \"O\"]}], \"words\": [\"Alice\", \",\", \"whose\", \"husband\", \"went\", \"jogging\", \"every\", \"Sunday\", \",\", \"liked\", \"to\", \"go\", \"to\", \"a\", \"dancing\", \"class\", \"in\", \"the\", \"meantime\", \".\"]}\n",
            "\n",
            "2020-12-20 09:08:55,842 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpmnefy_l0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hog7HwIHzdm8"
      },
      "source": [
        "Sample 5: The bright sun, the blue sky, the warm sand, the palm trees, everything round off."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6NFVmvYtzguX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48e86088-f681-4721-9ea4-42fd7476e6a8"
      },
      "source": [
        "!echo '{\"sentence\": \"The bright sun, the blue sky, the warm sand, the palm trees, everything round off.\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:08:58,132 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:08:58.293529: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:09:00,044 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:09:01,253 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:01,253 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:01,255 - INFO - filelock - Lock 140218919610240 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:01,255 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:09:01,255 - INFO - filelock - Lock 140218919610240 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:01,255 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:01,256 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpn3jl1yco\n",
            "2020-12-20 09:09:05,507 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:09:05,507 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpn3jl1yco/vocabulary.\n",
            "2020-12-20 09:09:05,507 - INFO - filelock - Lock 140218269110168 acquired on /tmp/tmpn3jl1yco/vocabulary/.lock\n",
            "2020-12-20 09:09:05,535 - INFO - filelock - Lock 140218269110168 released on /tmp/tmpn3jl1yco/vocabulary/.lock\n",
            "2020-12-20 09:09:05,536 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:09:05,536 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:09:05,536 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:09:05,536 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:09:05,536 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7f871c1b37b8>\n",
            "2020-12-20 09:09:05,537 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:09:05,537 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:09:05,537 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:09:05,837 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:09:05,837 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:09:06,048 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:09:08,747 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:09:08,748 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:09:08,748 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:09:08,748 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:09:08,748 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:09:08,748 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,749 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,750 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,751 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:09:08,752 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,753 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:09:08,754 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,755 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:09:08,756 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:09:08,786 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:09:08,787 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,787 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,787 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,788 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:09:08,789 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:09:08,790 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:09:09,268 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:09:09,268 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:09:09,268 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:09:09,268 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:09:09,268 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:09:09,269 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:09:09,269 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:09:09,269 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:09:09,269 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:09:09,561 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"The bright sun, the blue sky, the warm sand, the palm trees, everything round off.\"}\n",
            "prediction:  {\"verbs\": [], \"words\": [\"The\", \"bright\", \"sun\", \",\", \"the\", \"blue\", \"sky\", \",\", \"the\", \"warm\", \"sand\", \",\", \"the\", \"palm\", \"trees\", \",\", \"everything\", \"round\", \"off\", \".\"]}\n",
            "\n",
            "2020-12-20 09:09:10,283 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpn3jl1yco\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T9UCG-qN018X",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af4f9689-442f-4726-a908-7dba1a824036"
      },
      "source": [
        "!echo '{\"sentence\": \"The bright sun, the blue sky, the warm sand, the palm trees, everything rounds off.\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:09:12,636 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:09:12.789933: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:09:14,547 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:09:15,750 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:15,750 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:15,751 - INFO - filelock - Lock 139884787906432 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:15,751 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:09:15,751 - INFO - filelock - Lock 139884787906432 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:15,751 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:15,751 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpuj2lb1i1\n",
            "2020-12-20 09:09:19,983 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:09:19,983 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpuj2lb1i1/vocabulary.\n",
            "2020-12-20 09:09:19,983 - INFO - filelock - Lock 139884137381784 acquired on /tmp/tmpuj2lb1i1/vocabulary/.lock\n",
            "2020-12-20 09:09:20,009 - INFO - filelock - Lock 139884137381784 released on /tmp/tmpuj2lb1i1/vocabulary/.lock\n",
            "2020-12-20 09:09:20,010 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:09:20,010 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:09:20,010 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:09:20,011 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:09:20,011 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7f39504e07b8>\n",
            "2020-12-20 09:09:20,011 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:09:20,011 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:09:20,011 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:09:20,306 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:09:20,307 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:09:20,499 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:09:23,169 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:09:23,170 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,171 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,172 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:09:23,173 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,174 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:09:23,175 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:09:23,176 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,177 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:09:23,178 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:09:23,236 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:09:23,236 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,236 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,236 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,237 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:09:23,238 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:09:23,239 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:09:23,707 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:09:23,708 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:09:23,709 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:09:23,994 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"The bright sun, the blue sky, the warm sand, the palm trees, everything rounds off.\"}\n",
            "prediction:  {\"verbs\": [{\"verb\": \"rounds\", \"description\": \"[ARG1: The bright sun , the blue sky , the warm sand , the palm trees] , [R-ARG1: everything] [V: rounds] off .\", \"tags\": [\"B-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"I-ARG1\", \"O\", \"B-R-ARG1\", \"B-V\", \"O\", \"O\"]}], \"words\": [\"The\", \"bright\", \"sun\", \",\", \"the\", \"blue\", \"sky\", \",\", \"the\", \"warm\", \"sand\", \",\", \"the\", \"palm\", \"trees\", \",\", \"everything\", \"rounds\", \"off\", \".\"]}\n",
            "\n",
            "2020-12-20 09:09:24,932 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpuj2lb1i1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cBrxUvrL3Sp4"
      },
      "source": [
        "Sample 6 Ice pucks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rp77Vazw3QY8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8e32eb0-1162-4acc-976e-a25ee428dbda"
      },
      "source": [
        "!echo '{\"sentence\": \"Now, ice pucks guys!\"}' | \\\n",
        "allennlp predict https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz -"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-12-20 09:09:27,286 - INFO - transformers.file_utils - PyTorch version 1.5.1 available.\n",
            "2020-12-20 09:09:27.438284: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-12-20 09:09:29,226 - INFO - transformers.file_utils - TensorFlow version 2.4.0 available.\n",
            "2020-12-20 09:09:30,428 - INFO - allennlp.common.file_utils - checking cache for https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:30,428 - INFO - allennlp.common.file_utils - waiting to acquire lock on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:30,429 - INFO - filelock - Lock 139618002246904 acquired on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:30,429 - INFO - allennlp.common.file_utils - cache of https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz is up-to-date\n",
            "2020-12-20 09:09:30,429 - INFO - filelock - Lock 139618002246904 released on /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724.lock\n",
            "2020-12-20 09:09:30,429 - INFO - allennlp.models.archival - loading archive file https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz from cache at /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724\n",
            "2020-12-20 09:09:30,430 - INFO - allennlp.models.archival - extracting archive file /root/.allennlp/cache/e20d5b792a8d456a1a61da245d1856d4b7778efe69ac3c30759af61940aa0f42.f72523a9682cb1f5ad3ecf834075fe53a1c25a6bcbf4b40c11e13b7f426a4724 to temp dir /tmp/tmpewur_o27\n",
            "2020-12-20 09:09:34,712 - INFO - allennlp.common.params - type = from_instances\n",
            "2020-12-20 09:09:34,712 - INFO - allennlp.data.vocabulary - Loading token dictionary from /tmp/tmpewur_o27/vocabulary.\n",
            "2020-12-20 09:09:34,713 - INFO - filelock - Lock 139618002367880 acquired on /tmp/tmpewur_o27/vocabulary/.lock\n",
            "2020-12-20 09:09:34,741 - INFO - filelock - Lock 139618002367880 released on /tmp/tmpewur_o27/vocabulary/.lock\n",
            "2020-12-20 09:09:34,742 - INFO - allennlp.common.params - model.type = srl_bert\n",
            "2020-12-20 09:09:34,742 - INFO - allennlp.common.params - model.regularizer = None\n",
            "2020-12-20 09:09:34,742 - INFO - allennlp.common.params - model.bert_model = bert-base-uncased\n",
            "2020-12-20 09:09:34,742 - INFO - allennlp.common.params - model.embedding_dropout = 0.1\n",
            "2020-12-20 09:09:34,743 - INFO - allennlp.common.params - model.initializer = <allennlp.nn.initializers.InitializerApplicator object at 0x7efb596c07b8>\n",
            "2020-12-20 09:09:34,743 - INFO - allennlp.common.params - model.label_smoothing = None\n",
            "2020-12-20 09:09:34,743 - INFO - allennlp.common.params - model.ignore_span_metric = False\n",
            "2020-12-20 09:09:34,743 - INFO - allennlp.common.params - model.srl_eval_path = /usr/local/lib/python3.6/dist-packages/allennlp_models/structured_prediction/tools/srl-eval.pl\n",
            "2020-12-20 09:09:35,046 - INFO - transformers.configuration_utils - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /root/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n",
            "2020-12-20 09:09:35,047 - INFO - transformers.configuration_utils - Model config BertConfig {\n",
            "  \"architectures\": [\n",
            "    \"BertForMaskedLM\"\n",
            "  ],\n",
            "  \"attention_probs_dropout_prob\": 0.1,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_dropout_prob\": 0.1,\n",
            "  \"hidden_size\": 768,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 3072,\n",
            "  \"layer_norm_eps\": 1e-12,\n",
            "  \"max_position_embeddings\": 512,\n",
            "  \"model_type\": \"bert\",\n",
            "  \"num_attention_heads\": 12,\n",
            "  \"num_hidden_layers\": 12,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"type_vocab_size\": 2,\n",
            "  \"vocab_size\": 30522\n",
            "}\n",
            "\n",
            "2020-12-20 09:09:35,254 - INFO - transformers.modeling_utils - loading weights file https://cdn.huggingface.co/bert-base-uncased-pytorch_model.bin from cache at /root/.cache/torch/transformers/f2ee78bdd635b758cc0a12352586868bef80e47401abe4c4fcc3832421e7338b.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
            "2020-12-20 09:09:37,949 - INFO - allennlp.nn.initializers - Initializing parameters\n",
            "2020-12-20 09:09:37,949 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code\n",
            "2020-12-20 09:09:37,949 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.embeddings.LayerNorm.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.embeddings.position_embeddings.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.embeddings.token_type_embeddings.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.embeddings.word_embeddings.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.key.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.query.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.attention.self.value.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.0.output.dense.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,950 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.key.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.query.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.attention.self.value.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.1.output.dense.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.key.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.query.weight\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.bias\n",
            "2020-12-20 09:09:37,951 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.attention.self.value.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.10.output.dense.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.key.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.query.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.bias\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.attention.self.value.weight\n",
            "2020-12-20 09:09:37,952 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.11.output.dense.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.key.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.query.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.bias\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.attention.self.value.weight\n",
            "2020-12-20 09:09:37,953 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.2.output.dense.weight\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.bias\n",
            "2020-12-20 09:09:37,954 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.key.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.query.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.attention.self.value.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.3.output.dense.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.key.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.query.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.bias\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.attention.self.value.weight\n",
            "2020-12-20 09:09:37,955 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.4.output.dense.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.bias\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.key.weight\n",
            "2020-12-20 09:09:37,956 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.query.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.attention.self.value.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.5.output.dense.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.key.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.query.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.attention.self.value.weight\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,957 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.6.output.dense.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.key.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.bias\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.query.weight\n",
            "2020-12-20 09:09:37,958 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.bias\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.attention.self.value.weight\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,968 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.7.output.dense.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.key.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.query.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.attention.self.value.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,969 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.8.output.dense.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.output.dense.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.key.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.query.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.attention.self.value.weight\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.bias\n",
            "2020-12-20 09:09:37,970 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.intermediate.dense.weight\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.bias\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.LayerNorm.weight\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.bias\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.encoder.layer.9.output.dense.weight\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.bias\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    bert_model.pooler.dense.weight\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    tag_projection_layer.bias\n",
            "2020-12-20 09:09:37,971 - INFO - allennlp.nn.initializers -    tag_projection_layer.weight\n",
            "2020-12-20 09:09:38,432 - INFO - allennlp.common.params - dataset_reader.type = srl\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.lazy = False\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.cache_directory = None\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.max_instances = None\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.manual_distributed_sharding = False\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.manual_multi_process_sharding = False\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.token_indexers = None\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.domain_identifier = None\n",
            "2020-12-20 09:09:38,433 - INFO - allennlp.common.params - dataset_reader.bert_model_name = bert-base-uncased\n",
            "2020-12-20 09:09:38,744 - INFO - transformers.tokenization_utils - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n",
            "input 0:  {\"sentence\": \"Now, ice pucks guys!\"}\n",
            "prediction:  {\"verbs\": [], \"words\": [\"Now\", \",\", \"ice\", \"pucks\", \"guys\", \"!\"]}\n",
            "\n",
            "2020-12-20 09:09:39,466 - INFO - allennlp.models.archival - removing temporary unarchived model dir at /tmp/tmpewur_o27\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}